IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Application of: 

Prakash P. Mathew, et al. 

Serial No.: 10/723,033 

Filed: November 26, 2003 

For: IMAGE-BASED PATIENT 
DATA OBFUSCATION 
SYSTEM AND METHOD 



Commissioner for Patents 
P.O. Box 1450 
Alexandria, VA 22313-1450 



Sir: 

DECLARATION OF PATRICK S. YODER UNDER 37 C.F.R. § 1.131 

I, Patrick S. Yoder, hereby declare as follows: 

1. I am an attorney at the law firm of Fletcher Yoder in Houston, Texas. I am a 
registered patent attorney. My U.S. Patent and Trademark Office registration number is 
37,479. 

2. My business address is 7915 FM 1960 West, Suite 330, Houston, Texas 

77070. 



§ Group Art Unit: 2624 
§ 

§ Examiner: Le, Brian Q. 
§ 

§ Confirmation No.: 8841 
§ 

§ Atty. Docket: 133276-1 IT/YOD/LIU 
§ GEMS:0234 



I iieieb certify thai this coirespondence is tving tra niited t> ii 
the United States Patent and Trademark Office in accordance with 
C.F.R. § 1.6(d), or is being transmitted via the Office electronic fil: 
system in accordance with 37 C.F.R. § 1.6(a)(4), or is being deposited w 
the U.S. Postal Service with .sufficient postage i>\ first Class Mail in 
envelope addressed to: Commissioner for Patents, P.O. Box 1450, 
Alexandria, VA 22313-1450, on the date below: 



3. 



I prepared the above-identified patent application, and filed the same. 
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4. Prior to September 23, 2003, Fletcher Yoder was engaged by General 
Electric Company ("GE"), the assignee of the above-identified application, to prepare and 
file the above-identified application. This engagement is evidenced by an engagement letter 
sent to me by GE's Intellectual Property Department. A true and correct redacted copy of 
this engagement letter is attached hereto as Exhibit A. 

5 . Prior to September 23, 2003, 1 received a disclosure document pertaining to 
the above-identified application from GE's Intellectual Property Department. The 
disclosure document generally includes the subject matter disclosed and claimed in the 
above-identified application. A true and correct redacted copy of pages 1-7 of the 
disclosure document is attached hereto as Exhibit B. 

6. Upon the engagement of Fletcher Yoder, the above-identified application 
was duly docketed for preparation in the ordinary course of business. In the course of this 
engagement, a review of the disclosure materials and telephonic interviews were 
conducted with the inventors in the course of preparing the above-identified application 
for filing. Particularly, prior to September 23, 2003, 1 participated in a telephonic interview 
with Mr. Prakash Mathew, an inventor of record, to discuss the above-referenced disclosure 
document (Exhibit B). This telephonic interview is acknowledged by a true and correct 
copy of the redacted e-mail correspondence is attached hereto as Exhibit C. Additionally, 
as part of the patent preparation process, several email correspondences took place between 
myself and the inventors of record in an effort to prepare the above-identified application 
for filing. A selection of relevant e-mail correspondences is described below. 

7. On August 14, 2003, 1 received a status inquiry from Mr. Mathew with 
regard to the preparation of the above-identified application. A true and correct redacted 
copy of this correspondence is attached hereto as Exhibit D. This status inquiry was 
received after the telephonic interview acknowledged in Exhibit C. 
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8. The above-referenced disclosure document (Exhibit B) and the above- 
referenced interview (Exhibit C) clearly indicate that the inventors had conceived of the 
invention ultimately disclosed and claimed in the above-identified application and were in 
fact in possession of the invention at least as early as August 14, 2003. 

9. On August 19, 2003, 1 responded to Mr. Mathew's inquiry and informed 
him that 1 had not yet completed a draft of the above -identified application, but that I would 
try to complete the draft as soon as possible. This correspondence can be found in Exhibit 
D. 

10. On November 15, 2003, 1 sent an initial draft of the above-identified 
application to Mr. Mathew and the other inventors of record for review. I also sent a set of 
formal papers, including assignment and declaration papers, to be executed by the inventors. 
A true and correct redacted copy of this correspondence is attached hereto as Exhibit E. 

11. Upon receiving the executed formal papers, I filed the above-identified 
patent application on November 26, 2003, resulting in the constructive reduction to practice 
of the invention disclosed and claimed in the above-identified application. 

12. From prior to September 23, 2003 and through November 26, 2003, 1 had a 
backlog of case work for GE and other clients which prevented me from completing the 
draft of the above-identified patent application sooner. 

1 3. From prior to September 23, 2003 and through November 26, 2003, the 
diligent efforts contributing to the preparation and filing of the above-identified application 
were carried out in the ordinary course of business. 

1 declare further that all statements made herein are of my own knowledge, are true 
and that all statements made on information and belief are believed to be true, and further, 
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that these statements were made with the knowledge that willful false statements and the 
like so made are punishable by fine or imprisonment, or both, under 18 U.S.C. §1001, and 
that such willful false statements may jeopardize the validity of the application, any patent 
issuing thereon, or any patent to which this verified statement is directed. 



*h'J 0 * By: g§= 

Patrick S. Yoder 



Patrick S. Yodei 
Reg. No. 37,479 
FLETCHER YODER 
P.O. Box 692289 
Houston TX 77269-2289 
U.S.A. 



EXHIBIT A 



i 

j 




PRIVILEGED AND CONFIDENTIAL 



Patrick Yoder 

Fletcher, Yoder & Van Someren, P.C. 




RE: Disclosures for New Patent Applications 

Dear Patrick: 

Enclosed are copies of invention disclosures and inventor data sheets (as designated in the table 
below) for which a U.S. patent applications need to be drafted and filed as indicated. Please 




Remember that our fee arrangement includes all of the following items: 
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GE Docket #: 




Title: 




Designated Inventor: 




Assignee: 


GE Medical Systems Information Technologies, Inc. 






Target Filing Date: 




Foreign Filing: 




Special Comments: 





GE Docket #: 




Title: 




Designated Inventor: 




Assignee: 


GE Medical Systems Information Technologies, Inc. 






Target Filing Date: 




Foreign Filing: 




Special Comments: 





GE Docket #: 




Title: 




Designated Inventor: 




Assignee: 


Gl [edical Sys cms Inl rmation fechnologies In . 






Target Filing Date: 




Foreign Filing: 




Special Comments: 
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GE Docket #: 


_ 


Title: 




Designated Inventor: 




Assignee: 


GE Medical Systems Information Technologies, Inc. 






Target Filing Date: 




Foreign Filing: 




Special Comments: 





GE Docket #: 




Title: 




Designated Inventor: 




Assignee: 


1 IC. 






Target Filing Date: 




Foreign Filing: 




Special Comments: 





GE Docket #: 


133276IT/^CS:^3<' 


Title: 


Method and system for de-identifying burned in patient 
data from radiological images 


Designated Inventor: 


Yaseen Samara 


Assignee: 


GE Medical Systems Information Technologies, Inc. 






Target Filing Date: 




Foreign Filing: 




Special Comments: 
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GE Docket #: 




Title: 




Designated Inventor: 




Assignee: 


GE Medical Systems Information Technologies, Inc. 






Target Filing Date: 




Foreign Filing: 




Special Comments: 





When forwarding any billing relating to this matter please refer to the following billing codes: 




Legal IP Department 



i 

EXHIBIT B 



Crii ivieuicui systems 
QT Invention Disclosure Form 


Docket No.: 133276 


Mail to: PATENT OPERATION, W-71 0 








Date Received: BptSlllitl' 



Use as many pages in this word document as necessary. 

You may attach additional materials to support this disclosure, for example, Tech Notes and Drawings. 
Such submitted materials must be referenced in this disclosure form. Each page of these materials 
must be dated, signed and witnessed in the same manner as this invention disclosure. 



IIS 

INVENTION TITLE ; Provide a unique, descriptive title, if you vt/.y 'his disclosure in a language other than English, 
please provide a title in English as well Si you- >\-dig. : : en francos, mirci de proposer un litre en anglais et im litre en 
frangais. _ . ' 

Method and system for de-identifying burned in patient data from radiological images 

PW HiiXU B ACKi ,Ri > r\l); '■ \ ibe the nn Hem thai i ;o ved b die in e it m Assume t r tin t let has a baste 

Privacy laws within the medical practice are starting to be heavily enforced (if not already) and a number of regulatory laws 
mandate the restricted and appropriate disclosure of patient information in all medical records. Radiologists routinely need to, 
and have been transferring images to referring physicians and colleagues for consultation or teaching, as part of their 
workflow. Radiological images often contain patient identifying data (mainly text), which would allow a casual or informed 
observer to associate the image with an individual. Rather than change their established workflow, radiologists look to 
information system vendors such as GE to provide the tools to de-identify images of patient data to render them compliant to 
the regulatory law. 

Sometimes, the textual annotations are separate layers or tags within an imaging protocol such as DICOM. In such cases, 
suppressing the display of these tags alone, based on some coded set of rules would be a solution. This however is easy to do 
and intuitive. A bigger problem arises when the textual information identifying the patient is part of the image pixel set itself, 
i.e. it has been burned in. In such a case the de-identification problem for radiological images is not trivial and the solution of 
this would be a considerable benefit to the radiological community. 
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Vijaykalyan Yeluri 




X 






Denny Lau 




X 







= Primary Contact Inventor (to coordinate with Patent Evaluation Board and Preparing Attor 



2 WITNESS S ' I , Name Below) 


(Full Signature Below) 


DATE 















U.S.BETADisclForml/31/2000/revl 
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rvy FNT]y>\ SJf.?CTnPnnV 1 t how don ks and how it solves tki ^ T d abo 

As described earlier, the problem is that of automatically and conveniently de-identifying "burned in" patient information 

from radiological images. 

Lets clarify what this invention will address. . . 

The most common radiological image interchange method is using the DICOM protocol format. The image is stored and 
transported in the form of an electronic file. There could be patient identifying information in: 

A) The DICOM headers: This is a text portion of the file that encapsulates some metadata regarding the image. Not all 
information will be visible in the displayed image, but it still represents data that could potentially reveal the identity 
of the patient. The d e-identification method is programmatically coding which information tags will be deleted or 
retained in an image. I 



B) The DICOM overlay tags: These are also structured information that could be bitn 
image. They capture annotations, markers to regions of interests etc. | 



s that are laid over the actual 



The image pixel data. This is the actual pixel data with diagnostic information content. For convenience and record- 
keeping, patient identifying information is often burned into the exposure. This kind of patient data is hard to 
programmatically remove beca use there is no structured way of removing this data . The invention addresses this 
burned in data de-identification.^B^^^g^^^^^^^^^^^^^^^^^ 



The solution we propose will be the novel use of Optical Character Recognition (OCR) technology for the de-identification 
of radiological images. OCR technology itself is not new or novel and is quite a mature area. This is routinely and heavily 
used in the area of text scanning applications. The following is an excerpt from a casual search on the www.About.com 
website for the term Optical Character Recognition. 

OCR (Optical Character Recognition) is the process of turning a picture of words (such as a scan of 
a typed letter) into an editable document that you can open and use in your desktop publishing 
software, word processor, or other text editor. Today's OCR packages contain sophisticated support 
for multiple languages, PDF and HTML output, and format retention. 



|has Page Formatter to better retain formatting of pages and recognization of 

spreadsheets, tables, and other elements. Convert to and from PDF format. The International Voice 
Read Back reads your text aloud and now recognizes over 100 languages. 
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* = Primary Contact Inventor (to coordinate with Patent Evaluation Board and Preparing Attorney) 
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(Full Signature Below) 
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^retains page layout, allows output as Web pages, retains color in photos or graphics, and is 
Windows 2000 certified. This is a iow-cost OCR option. 



A powerful, BRurpnslngT^nexpensive alternative to some of the better known (and expensive) 
OCR packages. It reads 52 languages, preserves the page formatting, can scan color and dark 
backgrounds, has batch processing, and has expert tools for analyzing complicated layouts. 



4) 

p rc !fT^BjP^^^^^|j^^^5|M^j^^j, the Pro version is recommended for home use but 
is more advanced than the Home edition. Supports over 100 languages, has built-in spell-checker, 
handles PDF and HTML and retains document formatting. An under $100US OCR option. 



The Business Edition includes new and improved PDF Output, supports 56 languages, retains 
formatting, and outputs to HTML format. Includes scheduled batch processing for busy offices. 



The workflow involved might proceed something like this. 

1) A radiologist opens a study on a PACS system. He scans through the exams and selects one or more images for 
sending to another institution, lets say for example, teaching purposes. HIPAA mandates that patient demographic 
information should not be disclosed. 

2) The radiologist now selects the "De-identify including burned text" option in the PACS system. 



3) The PACS system now readies the images for sending out. It first negotiates the format of data it needs to send the 
images in. If DICOM, it creates a copy of the image file, removes the patient demographic information and any 
overlays in the header. If any other image format like GIF or JPEG, it deals only with the pixel data in the file. 
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In either case, the pixel data is fed into the OCR software program. This can be done progressively, m sections of 
the image, comprising a strip about 1cm x 5cm or some such small subset. Each section is quickly scanned and the 
OCR software determines if text is included in these regions. If text is found then an approximate position of the text 
character is returned. Instead of subsets of strips of the image, the entire image can be fed into the OCR software as 
well. 

The PACS system then creates an opaque mask and burns it into the pixels at the locations where text has been 
recognized. This intensity value of the opaque mask value can be pre programmed into the system. As long as the 
mask is a single intensity value, the character information underneath is blocked out. 

The modified image, with the masks burnt in and obscuring the text is then chosen as the image to be sent to the 
external system. The mask can be patterned or with a distinguishable border so as to clearly communicate to the 
viewer that information is missing from that region. 



:ompuur gc-WMiJ ,'lgurt: i/your inwntwn as you can twit e»;n 
twina to scale, but shvM be complete enough to show what you haw. 
m h v, ns ft ish a ra , // na} be used 



An example of a radiological image with patient identifying information is shown below: The patient n 
along with other textual annotations. 




After the steps outlined in the invention occur, the resulting image could be modified to look like one of the following: 
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In both cases, the textual information is overlaid out by a burned in mask, thereby anonymizing the identity of the patient 
involved. 

In a related embodiment of the invention, specific rules can be incorporated in the software to recognize and exclude certain 
useful text which do not identify a patient. Useful text is in the form of measurements or view descriptions such as rue phrase 
"FRONT" on the original image. Using this form of intelligent de-identification we could produce images like the following 



or like 





In this case, helpful information is retained while the patient identifying information is selectively masked out. 
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Block Diagram indicating for automatic de-identification 





No de-identification 
required for strip 
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There may be some cases, which will no doubt challenge the capacity of the invention. It will be cases where 

1 . The text is formatted vertically instead of the usual left to right approach. 

To address this problem ,we would configure the OCR software to read in vertical directions as well. This can be 
configured in addition to the left to right recognition. The user can choose to configure this or not, but the invention 
allows for this case. 



2. Lines running through the text could obscure the text or it could have extremely poor quality. 
For example, 









i 

i 
i 

i 

J 










i : 

I 1 ' 


m 






ADVANTAGES OK THE INVENTION : D, scribe the benefits of th 
newapplu tiot faster it igin & etc Jiufbusiness terms (e cost ran 

1 . Automatic: This de-identification is done without any manual intervention. 



Flexibility: The de-identification c 
determined rules. 



a be carried out at or in any subset of the image space, based on a set of pre- 



3 . Depending on the OCR software package used, a number of language texts can be recognized and removed. 

4. Security: Regulatory and safety guidelines and rules requiring image de-identification of patient information will be 
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EXHIBIT C 



From: Mathew, Prakash (MED) [IMCEAEX- 

_0=GE_OU=GEMSAM_CN=RECIPIENTS_CN=| 

Sent: j HB 

To: Pat Yoder 

Cc: Samara, Yaseen (GE Healthcare) 
Subject: RE: tcon to discuss 133276 and 133277 



Hi Pat. . 
Regarding docket application, 133276, here is an example of | 



It was a pleasure talking to you yesterday... 

Thanks 

__prakash 

Original Message ^^^^^^^^^ 

i i i H i ni i riiniii ly i * i fB' 

Sent: | 

To: Mathew, Prakash (MED); Samara, Yaseen (MED, GEMS-IT) 
Subject: RE: tcon to discuss 133276 and 133277 



he disclosures and \ 

time we can schedule more when we talk. 



Talk to you then! 



e can try to fit everything in the houi 



If we need more 



Original Message-- 

From : Mathew, Prakash (MED) [mailto f 
Sent: ■ 

To: 'Pat Yoder'; Samara, Yaseen (MED, GEMS-IT) 
Subject: RE: tcon to discuss 133276 and 133277 

How a bout 10:30-11:30 CST ? 
TCON ■|^HH#6726510 

_prakash 

Original Message-- 

From: Pat Yoder [mailto:| 
Sent: I ~"~ 

To: Samara, Yaseen (MED, GEMS-IT); Mathew, Prakash (MED) 
Subject: RE: tcon to discuss 133276 and 133277 



Guys, 



We have a regular meeting from 11:30 to about 1:00 i 
(Prakash, you picked my one window on this one.) 



n Thursdays. Can 'we work around that? 



Thanks. 



Original Message 

From: Samara, Yaseen (MED, GEMS-IT ) [mailto:9HHHHHHHH] 
Sent: gHSHHUHflffiBI 

To: Mathew, Prakash (MED); 'Pat Yoder' 
Subject: RE: tcon to discuss 133276 and 133277 



Sounds great! Thursday morning then. 

Original Message 

From: Mathew, Prakash (MED) 
Sent: ■^^■■^^■■HB 

To: 'Pat Yoder'; Samara, Yaseen (MED, GEMS-IT) 
Subject: RE: tcon to discuss 133276 and 133277 



Pat, Yaseen, 

I apologize for noUesponding earlier... 
How does Thu BH| morning sound? 
How about 

11:00 - 12:00 CST TCON f^HH #672651 0 {I'm flexible on 
the time that morning) 



prakash 



— Original Message-- 
From : Pat Yoder ["mailto:B 
Sent:[Z~™ 

To: Mathew, Prakash (MED); Samara, Yaseen (MED, GEMS-IT) 
Subject: tcon to discuss 133276 and 133277 



Guys, 

Any thoughts on this? I'm in all week, but probably need a couple hours 
advance notice before a tcon. 



Thanks! 



Pat 



Guys, 

As Yaseen and I discussed last week, we will be preparing patent 
applications on the referenced disclosures. I just returned from an extended 
trip and did review the disclosures during that time. I think the best way to 
proceed is through a telephone conference during which we can walk 
through the disclosures and I can flesh out points as needed to move along 
to the drafting stage. 

I will be in all week working on various projects. Could you please just let 
me know what works for you guys and I'll plan around that. I think we may 
need as much as a couple of hours to get through the material. One or both 
of you may with to participate, and I understand that Mathew will probably 
be the "point person" for these applications. 

Thanks, and I look forward to hearing from you. 



Pat 



Patrick S. Yoder 



Fletcher, Yoder & Van Someren 

Attorneys at Law 

7915 FM 1960 West, Suite 330 



Houston, Texas 77070 



tel: RJfvfef- •'^"•f 

fax: g^^^HBHHB 

CONFIDENTIALITY NOTE 

The documents accompanying this e-mail contains information from 
the law firm of Fletcher, Yoder and Van Someren which is confidential 
or privileged. The information is intended to be for the use of the 
individual or entity named in this transmission. If you are not the 
intended recipient, be aware that any disclosure, copying, distribution 
or use of the contents of this e-mailed information is prohibited. If you 
have received this e-mail in error, please notify us by telephone 
immediately so that we can arrange for the retrieval of the original 
documents at no cost to you. 



EXHIBIT D 



■ String E-mail Redacted - 



Sent: 




Sent; Tuesday, August 19, 2003 7:46 AM 
To: Mathew, Prakash (MED) 
Cc: Samara, Yaseen (MED, GEMS-IT) 
Subject: RE: Status of 133276 and 133277 ? 



Prakash, 

I have these laid out, but just have not completed them yet. Realistically, this will bleed over to next month. I'm 
personally working on these and will complete them as soon as I can. Please do not hesitate to follow up, but this 
work is on my very short list for completion. 



Thanks, 



-—Original Message 

From; Mathew, Prakash (MED) 

Sent: Thursday, August 14, 2003 6:10 PM 

To: 'Pat Yoder' 

Cc: Samara, Yaseen (MED, GEMS-IT) 
Subject: Status of 133276 and 133277 ? 
Importance: High 



Hi Pat, 

Just wondering about the status of the drafts for these cases... 

Thanks, 
_prakash • 

Ori ginal Message- 

From; j 
Sent: | 

To: Math ew, Prakash (MED); Samara, Yaseen (ME D, GEMS-IT) 
Subject: p — ■ = 



-Original Message— 
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• String E-mail Redacted - 



Pat Yoder 



From: Mathew, Prakash (MED) I 
Sent: Tuesday, December 02, 2003 12:26 PM 
To: Pat Yoder 

Cc: Samara, Yaseen (MED, GEMS-IT) 

Subject: RE: (GE docket 133276ITand 133277IT; our file GEMS:0234 and GEMS: 023 5) 



Thanks, 
_prakash 



Original Mess age- 

From: Pat Yoder | _ 
Sent: Wednesday, November 19, 2003 1:31 PM 
To: Samara, Yaseen (MED, GEMS-IT); Pat Yoder; Mathew, Prakash (MED) 
Subject: RE: (GE docket 133276ITand 133277TT; 'our file GEMS:0234 and GEMS: 023 5) 



-Original Message- 
From: Samara, Yaseen (MED, GEMS-IT) f 



Sent: Wednesday, November 19, 2003 12:16 PM 
To: 'Pat Yoder'; Mathew, Prakash (MED) 

Subject: (GE docket 133276ITand 133277TT; our file GEMS:0234 and GEMS:023 5) 



Pat, Prakash 



Original Mess aje- 

From: Pat Yoder | 

Sent: Saturday, November 15, 2003 5:50 PM 
To: 'Mathew, Prakash (MED) 1 
Cc: Samara, Yaseen (MED, GEMS-IT) 

Subject: draft patent application (GE docket 133276TT; our file GEMS:0234) 
Importance: High 



Prakash, 

As we discussed, we have proceeded to prepare a patent application for the referenced matter, 



o/onnno 
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relating to obfuscatlon of patient and other data from reconstructed images. Please find attached a 
copy of the application for your review and for review by the other co-inventors. Also attached are 
the Assignment and Declaration documents for review and signature by all inventors, I have also 
attached a .pdf file containing the drawings referred to in the application. Finally, I have attached a 
Notice regarding gur obligations to the Patent and Trademark Office whenever we file new patent 
applications. 

Please review the application for completeness and accuracy, and ask the other co-inventors to do 
the same. Please keep in mind the points brought out in the attached Notice^Onceyouhave 
reviewed the application, please call me with any questions or comments at£HHBS£HPlf you 
find the application complete and accurate, please print, sign and date the Assignment and 
Declaration documents, and have the other co-inventors do the same (noting.that the Assignment 
should be notarized when each of you signs it). \ 
Once signed PLEASE FAX THE ASSIGNMENT AND DECLARATION DOCUMENTS TO ME AT§HHR 
fHf (and retain a copy for your records-and in case we need them resent). I would also ask that 
you drop the documents in the mail to my office at the address below. We can file faxed copies, but 
must retain originals. IF NECESSARY, THE VARIOUS INVENTORS MAY SIGN DIFFERENT PHYSICAL 
COPIES OF THE DOCUMENTS. I will need all such copies faxed and mailed back to me, however. 

@BB9HBHBHHHHBHBBHBHHHM Accordingly, I would like to receive the signed 
documents for filing by MONDAY, NOVEMBER 24, 2003, AT THE, LATEST. 
Again, should you have, any questions or concern s, please do not hesitate to call or email me. 

Thanks again for your help and hard work: I look forward to your feedback. 
Pat 



Patrick S. Yoder 

FLETCHER YODER 
Attorneys at Law 
7915 FM 1960 West, Suite 330 
Houston, Texas 77070 
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The documents accompanying this e-mail contains information from the law firm of Fletcher 
Yoder which is confidential or privileged. The information is intended to be for the use of the 
individual or entity named in this transmission, if you are not the intended recipient, be 
aware that any disclosure, copying, distribution or use of the contents of this e-mailed 
information is prohibited. If you have received this e-mail in error, please notify us by 
telephone immediately so that we can arrange for the retrieval of the original documents at 
no cost to you. 



